The Internet has become a popular vehicle for sharing information. Using the Internet, a user has access to many millions of documents, such as Web pages, multimedia elements such as audio or audio/video files, images, and other information. Because locating information is problematic when such volumes of information are available, several Internet services provide a search capability. These services are commonly referred to as “search engines,” examples of which include AMERICA ONLINE (AOL) search, GOOGLE search, and LYCOS search. These search engines typically employ a “Web crawler” to retrieve Web pages and index text contained therein; store links to images, audio files, audio/video files, and other multimedia or other types of information that cannot be easily indexed; and discover additional pages to crawl or index by analyzing the content of the crawled pages. The links that search engines store can be uniform resource identifiers (URIs). Search engines may crawl some or all Web sites from time to time, such as periodically or when notified of new Web pages.
When a user desires to locate information in the Internet, the user may employ one of these search engines. The user can provide search text or other criteria and the search engine can return a list of links matching the provided search text or criteria. As an example, when a user enters “whales” as search text, the search engine may provide a set of links to Web pages containing the word “whales,” and also to multimedia elements having the word “whale” in their URI or other data associated with the multimedia elements.
Web pages sometimes contain links (e.g., URIs) to other documents or multimedia elements. As an example, a Web page relating to Alaska may contain links to other Web pages containing information about gray whales, Orca killer whales, and grizzly bears. A Web crawler that crawls and indexes the Web page relating to Alaska may add the links to the other Web pages to its queue of Web pages that are to be crawled and indexed. Thus, the crawler can then crawl and index the Web pages relating to gray whales, Orca killer whales, and grizzly bears. These Web pages may describe these creatures and provide pictures or other multimedia elements, such as videos, relating to these creatures. These Web pages may also have additional links to other Web pages containing additional information about these or other creatures. The Web crawler may iteratively crawl and index these additional links. Thus, a Web crawler can discover additional Web pages for crawling and indexing by following links contained in Web pages.
Web pages can generate links dynamically, such as by using a client-side script. A script is a set of instructions that a computer can execute. A script that is designed to execute at a client computing device rather than a server computing device is called a “client-side script.” Web pages can employ various programming languages for client-side script, such as JAVASCRIPT, JSCRIPT, VBSCRIPT, and so forth. The script may generate a link based on various variables or user actions. As an example, a Web page may provide a list of animals. When a user selects one of the animals, a client-side script contained by the Web page may dynamically generate a link (e.g., URI) based on the selected animal and display a multimedia element featuring the selected animal. Because client-side scripts typically cannot be executed by a server computing device outside the presence of a Web browser, conventional Web crawlers are unable to locate dynamically generated links, and are therefore unable to identify, index, or crawl documents located at these dynamically generated links.